The Burrows-Wheeler Transform for Block Sorting Text Compression: Principles and Improvements

A recent development in text compression is a “block sorting” algorithm which permutes the input text according to a special sort procedure and then processes the permuted text with Move-to-Front and a final statistical compressor. The technique combines good speed with excellent compression performance. This paper investigates the fundamental operation of the algorithm and presents some improvements based on that analysis. Although block sorting is clearly related to previous compression techniques, it appears that it is best described by techniques derived from work by Shannon in 1951 on the prediction and entropy of English text. A simple model is developed which relates the compression to the proportion of zeros after the MTF stage. Short Title Block Sorting Text Compression Author Peter M. Fenwick Affiliation Department of Computer Science The University of Auckland Private Bag 92019 Auckland, New Zealand. Postal Address Dr P.M. Fenwick Dept of Computer Science The University of Auckland Private Bag 92019 Auckland New Zealand. E-mail [email protected] Telephone + 64 9 373 7599 ext 8298